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ABSTRACT 


Deep Reinforcement Learning has led us to newer possibilities in solving 
complex control and navigation related tasks. The paper presents Deep 
Reinforcement Learning autonomous navigation and obstacle avoidance of self¬ 
driving cars, applied with Deep Q Network to a simulated car an urban 
environment. "The car, using a variety of sensors will be easily able to detect 
pedestrians, objects will allow the car to slow or stop suddenly. As a computer is 
far more precise and subject to fewer errors than a human, accident rates may 
reduce when these vehicles become available to consumers. This autonomous 
technology would lead to fewer traffic jams and safe road". 
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INTRODUCTION 

The autonomous car has been in the headlines for a decade 
and still continues to dominate auto headlines. The 
autonomous car has attracted the researchers, robotics 
communities and the automobile industries. Human driving 
is accident-prone. We are drunk, drowsy and tired. Our 
inefficiency to make better on-road decisions cause road 
accidents, property loss and deaths. The autonomous car 
brings us the possibilities to replace accident-prone human 
driver by providing comfort and safety. An autonomous car 
is a vehicle that can guide itself without human interaction. 
This kind of vehicle has become a concern reality and may 
pave the future system where computers take over the art of 
driving. An autonomous car is also known as driverless car, 
robot car, self-driving car or autonomous vehicle. 
Autonomous cars uses various kinds of technologies. They 
can be built with GPS sensing knowledge to help with 
navigation. They may use sensors and other equipment to 
avoid collision. They also have the ability to use a range of 
technology known as augmented reality, where a vehicle 
displays information to drivers in new and innovative ways. 
Autonomous car production could cause problems with 
existing auto insurance and traffic controls used for human- 
controlled cars. It is only a matter of time before these kinds 
of advances allow us to outsource our daily commute to a 
computer. The automated car concept was originated in 
1920, but it evolved truly in 1980 with CMU's Navlab and 


ALV projects. CMU's effort resulted in the ALVINN in 1989, 
the first try in car autonomy which used a neural network 
to drive. The United States government invested in three 
projects for autonomous car which were demonstrated by 
US Army (Demo-I, Demo-III) and DARPA (Demo-II) in the 
2000s. From 2004 to 2007, DARPA organised 3 car 
autonomy challenges which got attention ah over the world 
at that time. The automobile industries like Ford, Mercedes, 
Volkswagen, Audi, Nissan, Toyota, BMW and Volvo came to 
play a bigger role in the autonomous car area by announcing 
their plans for it. In 2014, Google introduced Google car with 
sensor fusion technology (mainly camera and lidar sensor). 
One year later, Tesla brought their autonomous car in 2015. 
Nvidia has recently introduced a self-driving car with end-to- 
end convolutional neural network. In 2017, Audi stated 
"latest A8 would be autonomous at up to speed of 60 km/h 
using Audi AI. The driver doesn't require safety checks such 
as frequently gripping the steering wheel." 

The Audi A8 is claimed to be the first production car to reach 
level 3 autonomous driving and Audi would bethe first 
manufacturer to use laser scanners in addition to cameras 
and ultrasonic sensors for their system. On the 7 th November 
2017, Waymo announced that it has begun testing driver¬ 
less cars without a safety driver at the driver position. 
However, there is an employee in the car. And the list still 
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continues with Uber, Tata, Intel and other automobile 
industries. Driving is a complex problem. The evolution of 
autonomous car involves complex modules, which uses 
traditional control algorithms , SLAM based navigation 
systems and Advanced Driver Assistant Systems in recent 
days. But all the previous methods fail to replace human 
driver completely. But increasing computing power of GPUs, 
accurate sensors and deep neural networks has made 
complete autonomy is realisable. But recent failure of Uber 
and Tesla, and the listed disengagements in show that the 
autonomous car research needs more attention in the 
upcoming years to achieve 'better than human-level driving'. 
In the context of 'human-level control', presented a deep 
reinforcement learning approach which redesigned Q- 
Learning using a deep neural network. It introduces 4 steps 
to unstable Q-learning: Experience Replay, Target Network, 
Clipping Rewards and Skipping Frames. It achieved human- 
level performance (sometimes even better than human) 
tested in different complex game environment. The 
breakthrough achievement of DQN has motivated us to 
implement it in car autonomy for human-level performance. 
In this paper, we present a DRL-based autonomous driving 
strategy for urban environment using Deep Q Network. We 
use sensor fusion on two types of input sensor data: vision 
based camera sensor and a laser sensor. The approach is 
tested in simulation environment, which is designed using 
Unity Game Engine. We also design a car prototype capable 
of running our approach in real-time. The prototype uses 
Hokuyo Lidar Sensor and camera sensors embedded with 
Nvidia Jetson TX2 embedded GPU. The paper structure is as 
follows: it introduces autonomous car and the neural 
network approaches in car autonomy, presents the 
background and mathematical representation of Deep Q 
Network and describes proposed methodology with the 
network architecture and configurations, action definitions 
and simulation environment. It also presents the learning 
curve during network training. Section IV discusses our car 
prototype capable of running DQN in real-time. 

I. PROPOSED METHODOLOGY 
A. Network Architecture 

We design a deep neural network based on fully-connected 
CNN to approximate Q-function. The network consists of 3 
Convolutional layers and 4 dense layers. Two types of data 
Input: front camera image (RGB image of size 80 x 80) and 
lidar data (grid-map of size 80 x 80) are fed into the neural 
network, processed using the convolutional layers data type. 
Two data types are fused using a flatten layer, preceded by a 
dense layer and a pooling layer (used for data resize) for 
each data type. Our target network follows the same 
architecture, which is updated every 6000 steps. The neural 
network is trained to accept 5 actions with defined reward. 
The network uses mini-batch of size 32, trained with 
learning rate 0.001 
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Fig: 1 System Architecture 


B. Action and Reward Definition 

Action definition is one of the key points in Reinforcement 
based learning strategy. We choose 5 actions for the agent to 
be trained with DQN: Keep Going (reward^ speed/5), Left 
(reward = -0.6), Right (reward = -0.2), Accelerate (reward = 
+ 1) and Brake (reward = -0.4). The basic action for our agent 
is to keep going (or in other terms do nothing) in the road. 
For a hit, a bad reward of value - 6 is assigned. It needs to 
learn how to accelerate when the front is empty and 
decelerates (brake) when there are other cars or agents in 
front 
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C. Simulation Environment 

We design a simulated urban environment in Unity Game Engine. The 
environment is based on Geneve environment and consists of 5 lanes 
highways. It is a city-like structure (see fig. 3) with buildings, trees, and 
street lights. The environment is cooperative; other cars run on the 
highways (see fig. 4). The basic goal in our design is to mimic the real¬ 
time urban environment as close as possible in the simulation, hallows 
us to train the DQN to drive the car autonomously, with almost real-time 
driving behavior _ 



D. Software Architecture 

The simulation test is done in a Ubuntu platform (v. 16.04LTS) 
installed with python 3 (v. 3.6.2). The Deep Q Network is designed 
using tensorflow API. OpenCV (v. 3.3.0) is used for image related 
tasks. It uses numpy, scipy and matplotlib. The simulator is connected 
to the DQN algorithm using Socket connection (python-socket-10). Figure 5 
illustrates the software used in simulation 
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E. OUR REAL-TIME CAR PROTOTYE 

Basedonthesimulationset-up,wedesignedacar capable ofrunningDeep 
Neural Networks in real-time. Deep Neural Networks are data-centric. The 
performance of such methods depends on the computing power of the system. 
We consider Traxxas Remote Control Racing car and redesign it with 
Nvidia-Jetson TX2; the fastest embedded GPU. We add a Hokuyo lidar 
sensor (model: Hokuyo UTM-30LX Scanning Laser Rangefinder) and a 
camera. Ubiquity wifi access point provides the car with required 
communication to the ground station. Figure presents our car prototype 
capable ofrunning DQN algorithm in real-time with sensor data fusion. Figure 
9 presents our designed car prototype capable of running DQN in real-time. 
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